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ABSTRACT 


Little is known about the influence of various sensory modali- 
ties such as taste, smell, color, and thermal, towards perceiving 
simulated flavor sensations, let alone their influence on people's 
emotions and liking. Although flavor sensations are essential in 
our daily experiences and closely associated with our memories 
and emotions, the concept of flavor and the emotions caused by 
different sensory modalities are not thoroughly integrated into 
Virtual and Augmented Reality technologies. Hence, this paper 
presents 1) an interactive technology to simulate different flavor 
sensations by overlaying taste (via electrical stimulation on the 
tongue), smell (via micro air pumps), color (via RGB Lights), and 
thermal (via Peltier elements) sensations on plain water, and 2) a set 
of experiments to investigate a) the influence of different sensory 
modalities on the perception and liking of virtual flavors and b) 
varying emotions mediated through virtual flavor sensations. Our 
findings reveal that the participants perceived and liked various 
stimuli configurations and mostly associated them with positive 
emotions while highlighting important avenues for future research. 
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Figure 1: Two virtual flavor apparatus (Left: virtual coffee 
mug and Right: virtual beverage glass): (a) thermal and smell 
stimuli delivery, (b) & (d) electronic components and smell 
containers, (c) color and smell stimuli delivery. 


1 INTRODUCTION 


The concept of flavor is richly used in our everyday lives. Flavor is 
a multimodal sensation that is mainly determined by the taste and 
smell modalities [2]. Additional modalities such as color, texture, 
temperature, and sound also contribute towards the perception 
of flavor sensations. It is known that the flavors in our foods and 
beverages evoke vivid, colorful memories and emotions. Recent 
advancements in neurobiology and other related fields also reveal 
close associations of human olfaction and gustation with our mem- 
ory and emotional responses [27]. Due to this entwined nature of 
flavor perception and emotional responses, flavors immediately 
remind us of a joyful holiday, being ill, or other past positive or 
negative memories. For example, we prefer to have a cup of tea or 
coffee when we are tired, eat candies when we are stressed, and 
sip a glass of lemonade on a hot summer day to relax - all these 
experiences are related to our emotions [25]. However, the contri- 
butions of various sensory modalities, including taste, smell, color, 
and thermal, towards flavor perception and the human emotions 
mediated through various modalities remain underexplored. 


To study the effects of different sensory modalities on flavor per- 
ception, in this paper, we 1) present a novel approach to formulate 
flavor sensations by overlaying taste, smell, color, and thermal stim- 
uli on plain water (i.e. Virtual Flavors), 2) investigate the influences 
of these multimodal stimuli on the perception of virtual flavors in 
terms of four primary taste sensations (i.e. salty, sweet, bitter, and 
sour), and 3) examine how different modalities affect participants 
emotions and liking towards a simulated flavor sensation. 

As shown in Figure 1, we have developed two apparatuses, to 
deliver virtual flavor sensations: 1) a virtual beverage glass to simu- 
late common beverages such as soft drinks by manipulating electric 
taste (electrical stimulation on the tongue via silver electrodes at- 
tached on the rim of the glass), smell (dispersed on the surface of 
the beverage via micro air pumps and liquid state essential oils), and 
color (projected on the beverage via RGB Light Emitting Diodes) 
sensations, and 2) a virtual coffee mug to simulate coffee beverage 
experience based on smell (dispersed on the surface of the beverage 
via micro air pumps and liquid state essential oils), and thermal 
(via Peltier elements attached on the disposable coffee mug) sensa- 
tions. Our technology’s working principle is to overlay different 
virtual stimuli (applying externally) when drinking from these two 
apparatus, enabling users to experience different virtual flavors. 

Different sensory modalities are featured in two apparatus to 
simulate virtual beverage experiences. In virtual beverage glass, 
when a user drinks from the mouthpiece, colors are projected onto 
the beverage (visual), smells are released on the surface of the 
beverage (olfaction), and sour and salty sensations are applied via 
electrical stimulation (gustation) thus simulating common sensory 
modalities in everyday mixed drinks. Similarly, in the virtual coffee 
mug, virtual flavors are delivered through scents dispersed near 
the user’s nose (olfaction) and thermal stimuli (hot and cold tem- 
peratures) near the user’s upper lips (to simulate temperatures of 
coffee). Users’ perception of single or combined stimuli creates 
virtual flavor sensations external to the plain water they consume 
using both apparatus, which is a novel aspect of this research. 

A user experiment is conducted using both apparatus to apply 
different electric taste, smell, color, and thermal stimuli over plain 
water at room temperature. Participants were instructed to use both 
devices to consume virtually flavored water and provide their feed- 
back on perceived virtual flavor sensations based on four primary 
taste sensations, their liking towards simulated sensations, and the 
emotions mediated through these flavor stimuli. Thus, key contribu- 
tions of this paper are: 1) examining a wide range of digital stimuli 
configurations and specific changes in flavor perception with re- 
gards to four primary taste sensations salty, sour, bitter, and sweet, 
2) studying participants’ hedonic liking towards various stimuli 
configurations, 3) studying the emotions mediated through differ- 
ent stimuli configurations, and 4) introducing a novel approach to 
implement digital flavor experiences in future AR and VR technolo- 
gies. Besides, the work presented in this paper contributes towards 
achieving the digital controllability of flavor sensations. 


2 LITERATURE REVIEW 


The perception of flavor is a multisensory experience that involves 
information from an individual or varying combinations of basic 
human senses (i.e., sight, sound, taste, smell, and touch) [2]. Even 


though the prior experiences and expectations are involved in flavor 
perception, it is commonly narrowed down to the intrinsic flavor 
components, which are the sensations of taste and smell [22]. The 
co-exposure of taste sensations (known as salty, sour, sweet, bitter, 
and umami) together with numerous smell sensations generally 
provide a wholesome sensory experience of the flavor [37]. Several 
prior works have explored the smell - taste interactions and their 
congruence in flavor perception; however, it remains uncertain how 
the overall flavor is perceived as a function of taste, smell, and other 
sensations [14, 29, 47, 50]. 

Apart from smell and taste sensations, the flavor perception is 
considerably influenced by the information from other sensory 
channels such as the vision, texture (mouthfeel), temperature, and 
the look and shape of eating or drinking utensils [9, 46]. Due 
to cross-modal interactions between colors and previous experi- 
ences with food and beverages, upon seeing them, we create a 
pre-perception of flavor in our mind based on their attributes such 
as color, presentation, fizziness, and temperature. Studies conducted 
on cross-sensory effects on visual stimuli and flavor interpretation 
(e.g., the interactive pseudo-gustatory display that superimposes 
virtual colors on to a drink [32]) highlight that different colors may 
induce different flavors regardless of the beverage presented. 

In recent years, there has been a growing interest in the simula- 
tion of flavor sensations in VR, HCI [16] and consumer products [10]. 
Research works proposed by Ranasinghe et al. and Nakamura et al. 
demonstrate using controlled electrical stimuli to simulate primary 
taste sensations [30, 31, 38, 39, 41, 42]. In [41], the authors presented 
an experiment to study familiar and unfamiliar types of pairings 
that can be formed between smell and electric taste stimuli. The 
results highlight that matching smell - taste pairs increase the per- 
ception of single taste sensations (e.g., sour taste and lemon smell 
increase the perception of sour sensation) while non-matching pairs 
increase the perception of several taste sensations (e.g., sour taste 
with chocolate smell increases the perception of salty, sour, and 
sweet sensations). Additionally, Nakamura et al. applied electric cur- 
rent stimuli through isotonic drinks (which contains electrolytes) 
and food (vegetables, fruits, and cheese) to change their taste per- 
ceptions by varying voltage levels [30]. In [31], they have also 
measured the latency between presentation and perception using 
visual and electric taste stimuli. 

Liking towards certain virtual flavor stimuli and mediated emo- 
tions from virtual flavor stimuli remain unexplored in VR and HCI 
research, especially when coupled with electric taste sensations. 
People’s hedonic liking towards a flavor sensation is mainly based 
on its smell and taste sensory impressions. In addition, previous or 
repeated exposure, pleasant or unpleasant memories related to a 
specific flavor, ambience are also some of the factors affecting the 
liking towards a flavor sensation. As explained in [28], people also 
like certain flavors due to the number of calories associated with 
the food or drink. For example, although beverages such as beer 
and coffee taste bitter, people still develop higher preferences due 
to the calories consumed and smells released. This phenomenon is 
commonly known as the ‘associative learning of flavors’ where the 
liking responses are increased when the taste sensations are paired 
with smell sensations [11]. 

Flavors have a strong influence on people’s emotions and vice 
versa - flavors can regulate and alter our emotions, at the same time, 


Figure 2: Different components of (A) virtual coffee mug, 
and (B) virtual beverage glass: (a) micro-air pumps, (b) smell 
containers, (c) main control module, (d) thermal module, (e) 
disposable coffee cup, (f) RGB color module and smell dif- 
fusing tube, and (g) electric taste module. 


emotions can influence our food choices and appetite [3, 26, 26, 35]. 
However, in VR and HCI research, mediated emotions through fla- 
vor sensations are not studied in detail, let alone the influence of dif- 
ferent flavor attributes towards people’s emotions. Recently, Gayler 
et al. present an exploratory study on the relationship between the 
taste of 3D printed food and their emotional experiences [15]. Their 
findings indicate that the taste-emotional valence mapping (sweet 
as a positive and bitter as a negative emotion) extends into real-life 
scenarios, and the taste-emotional arousal mapping is proportional 
between each other (higher the taste higher the emotions). 

Measuring emotions in an interactive system is uniquely chal- 
lenging as people’s emotions vary based on multiple factors, such 
as the context of use and difficulties in accurately verbalizing spe- 
cific emotions. There exist many scales, such as Self-Assessment 
Manikin(SAM) [6], Emotion wheel [43], and EsSense Profile [33, 34], 
as well as many vocabulary terms to measure different emotional 
conditions. In this paper, we adopt the EsSense Profile to measure 
participants’ emotions [19] as it is widely accepted in measuring 
emotions associated with foods and beverages, especially in sensory 
sciences and studies [17, 20]. 


3 APPARATUS DESIGN AND DEVELOPMENT 


The working principle behind both virtual beverage glass and vir- 
tual coffee mug is the same. When a user takes a sip from either of 
these apparatus, the smell, taste, color, and thermal stimuli will be 
applied externally to enhance the base drink’s flavors. The virtual 
beverage glass has three subsystems, as shown in Figure 2 (B): the 
RGB color projection module, the electric taste module, and the 
smell module. The virtual coffee mug has two subsystems, as illus- 
trated in Figure 2 (A): the smell module and the thermal module. 
DFRobot’s Arduino Bluno Nano ! controls different subsystems in 
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Figure 3: A participant interacts with the (a) virtual coffee 
mug, and (b) virtual beverage glass. 


both apparatus. Various combinations of stimuli can be formulated 
and activated using a computer or a smartphone via Bluetooth. 


3.1 The RGB color projection module 


To simulate colored beverages, an RGB LED is attached to the virtual 
beverage glass (as in Figure 2 (B) (f)) that projects distinct colors 
based on different stimuli configuration. 


3.2 The electric taste module 


A constant-current source is developed in the virtual beverage glass 
to apply weak and controlled electrical pulses to the user’s tongue 
while drinking, thus simulating sourness and saltiness as explained 
in [39, 40]. The constant-current source ensures that the electrical 
stimuli are not altered depending on the resistance of the user’s 
tongue [21, 39]. Electrical stimuli are presented via two 95% silver 
electrodes attached to the rim of the virtual beverage glass. Electric 
sour and salty stimuli are formed as below: 


e Sour: magnitude of current: 1804A, frequency: 800Hz 
e Salty: magnitude of current: 404A, frequency: 400Hz 


3.3 The smell modules 


The smell modules in both apparatus are implemented using three 
components: the micro air-pumps, smell containers that hold essen- 
tial oils, and polyurethane tubes. Each air-pump can be actuated 
individually, forcing air into a specific smell container. Each con- 
tainer has essential oil-soaked cotton balls that acted as a medium 
to hold the particular scent consistently. In the virtual beverage 
glass, lemon and chocolate scents, as well as a spirit-like scent 
(prepared by mixing lime, cognac, and isopropyl alcohol), is used, 
while coffee, chocolate, and mint essential oils are used in the vir- 
tual coffee mug. The orthonasal olfaction approach is employed 
to deliver smell stimuli to the user’s nose. The intensity of smell 
stimuli is regulated using a Pulse-Width Modulated (PWM) control 
algorithm to operate air-pumps in specific paces. All the essential 
oils are selected from Aftelier’s chef’s essence flavor products’. 


“https://www.aftelier.com/ 


3.4 The thermal module 


In the virtual coffee mug, a Peltier element attached to a silver 
metal piece is used to apply controlled thermal stimuli on or near 
the user’s upper lips and nose simulating hot or cold coffee expe- 
riences. A PWM control algorithm is implemented to manage the 
intensity of the thermal stimuli. The direction of the Peltier ele- 
ment (hot vs. cold) is controlled via a motor driver, which consumes 
approximately 7 Watts. A heat sink is attached to the Peltier mod- 
ule to extract the excessive heat generated on the Peltier element 
over time. Additional safety mechanisms are also implemented to 
confirm the temperatures applied are within a comfortable range. 


4 EXPERIMENTAL SETUP AND EVALUATION 


The primary purpose of the experiment presented in this paper 
was to investigate the influence of external sensory stimuli on par- 
ticipants’ likeness and emotion-profile while consuming different 
virtual beverages, thus validating and supporting the concept of 
virtual flavor technology. The study presents two apparatus to sim- 
ulate real-world experiences: 1) Virtual beverage glass that mimics 
the mocktail or cocktail glassware through which the liquid is visi- 
ble. 2) Virtual coffee mug, which is to imitate a coffee mug or travel 
mug through which the liquid inside the cup is invisible. 


4.1 Participants 


A sample set of 30 healthy people were chosen for the study with 
both the apparatus. They consumed both virtual beverages and vir- 
tual coffee flavors. One participant was unable to return to consume 
the virtual coffee; therefore, the participant demographics are as 
follows. The experiment was conducted with 30 participants for the 
virtual beverage glass (17 male and 13 females) in the age range of 
18-37 years (M = 26.42, SD= 4.8), and with 29 participants (17 males 
and 12 females) in the age range of 18- 40 years (M = 22.89, SD= 5.23) 
for the virtual coffee mug. They were instructed to refrain from 
consuming any food or beverages with heavy flavors such as coffee 
or spicy food an hour before the experiment. Furthermore, they 
were asked not to wear any cologne during the study and while 
attending their sessions to minimize external effects on the stimuli 
configurations. The participants gave their consent before the study, 
and each session took approximately 20 minutes to complete. 


4.2 Stimuli configurations 


The study was conducted in an air-conditioned research laboratory 
with minimum interference. A quiet, odor-free environment was 
maintained to ensure a controlled testing environment. None of 
the people entering the room were allowed to wear any fragrance 
products during the study. The minimum time gap between each 
participant was 30 minutes, which was adequate to reduce the 
lingering smells in the room. Bottled-water that is taste and odor- 
free was utilized to maintain the uniformity of the base drink, 
and avoiding any influence on the participants’ flavor and emotion 
perceptions. The devices were sanitized using 70% isopropyl alcohol 
wipes before and after each testing session. 


4.3 Virtual beverage glass 


There were 11 different stimuli configurations in four different 
modalities for the virtual beverage glass, as described in Table 1. 


Table 1: Stimuli configuration for virtual beverage glass. 


Modality 
Green 


Taste fse fso | 


Combinations | Lemon + Sour | Chocolate + | Spirit + Sour + 
+ Yellow Salt + Red Green 
Table 2: Stimuli configuration for virtual coffee mug. 
Modality 
“Temperature [Hot [oa | 


Combinations | Hot + Coffee Hot + Choco- | Hot + Mint 
late 
a Cold + Coffee | Cold + Choco- | Cold + Mint 
late 


Different stimuli configurations were selected based on prior work 
and to cover various combinations of visual, electric taste, smell, and 
thermal sensations. When choosing color stimuli, special attention 
was given to select natural colors from common food and beverages, 
in addition to the visibility of specific colors through plain water 
and glassware. For example, the blue color is not selected as it is less 
favored in food products. The color blue is considered an atypical 
food color as there are very few naturally occurring blue colored 
food [36]. Saltiness and sourness were selected as they are the two 
prominent taste sensations reported via electrical stimulation on 
the tongue [30, 39, 40]. Familiar smells such as lemon, chocolate, 
and spirit-like (alcohol-like) essential oils were used to formulate 
smell stimuli. 


4.4 Virtual coffee mug 


There were 11 different stimuli configurations in three different 
modalities for the virtual coffee mug, as described in Table 2. Famil- 
iar smells such as coffee, chocolate, and mint were used with hot 
and cold thermal sensations in the virtual coffee mug. 

We have limited the number of possible conditions in the "combi- 
nations" category due to too many potential candidates. We noticed 
that participants were tired during our preliminary experiments 
due to repeated exposure when the experiment was too long. Fur- 
thermore, prolonged exposure, even with longer intervals between 
stimulus, generally results in sensory habituation (decreasing re- 
sponsiveness to stimuli). 


4.5 Experimental Method 


During the study with virtual beverage glass, the participants were 
allowed to fill their apparatus with water as and when they needed 
between the different samples. For the virtual coffee mug, one 
water-filled cup was enough for a participant. Participants were 
informed that they are drinking plain water using both apparatus as 
per the Institutional Review Board (IRB) approval. The presentation 
order of different samples was counterbalanced to avoid the order 
bias. After each sample, participants were asked to complete a 


questionnaire to record perceived flavor sensations (in terms of 
four primary taste sensations), liking towards the stimuli (overall 
liking, smell liking, taste liking, color liking, and temperature liking), 
and emotions mediated through the stimuli. 

The flavor sensations were recorded after each stimulus in terms 
of salty, sour, sweet, and bitter sensations on the JAR (Just about 
right) scale [22]. JAR Scale is commonly used to evaluate the appro- 
priateness of the level of a particular attribute in a product using a 
five-point categorical scale ranging from ‘much too little’ to ‘much 
too strong’. For example, the JAR scale determines whether a spe- 
cific taste sensation is too high, too low, or just about right in a 
particular virtual flavor configuration. They have also rated the 
liking towards each stimulus on a 9-point hedonic scale, which 
ranges from ‘extremely dislike’ to ‘extremely like’ [22, 24]. 

Moreover, for each stimulus, participants were asked to check 
all the emotions that they can relate to on the check-all-that-apply 
(CATA) scale [1, 34]. As described in [13], the CATA approach is 
considered one of the quick and straightforward methods to gather 
the participants’ emotions towards the flavor combinations while 
requiring little cognitive effort as they select all that applies related 
to a particular stimulus. Thus, the CATA scale was incorporated 
to evaluate participants’ emotions, and the list of emotions was 
provided based on the EsSence Profile following [18]. Composite 
positive and negative emotion scores were created by taking the 
sum of the positive emotions and the negative emotions, as shown 
in [34]. We employed a similar questionnaire format to record 
participants’ feedback keeping both studies consistent. 


5 RESULTS 


The main findings of the user experiment using two apparatus (vir- 
tual beverage glass and virtual coffee mug) are discussed in this 
section. Participants’ responses concerning different virtual flavor 
stimuli and mediated emotions are recorded in a similar manner 
employing both apparatus, while the virtual beverage glass pro- 
vided feedback on the overall liking, smell liking, taste liking, and 
color liking and the virtual coffee mug provided feedback on overall 
liking, smell liking, taste liking and temperature liking based on 
different modalities included in each apparatus. In each condition, 
repeated measures ANOVA were conducted to determine signifi- 
cant differences based on p-values < 0.05 and conducted follow up 
post hoc analysis where necessary. 


5.1 Results: Virtual Beverage Glass 


5.1.1 Grouping the Virtual Beverage Glass Modalities: The stim- 
ulus combinations for the virtual beverage were combined into 
four different modality groups. The groups are as follows, color 
(containing the red, green, and yellow color stimuli), electric taste 
(containing the salty and sour stimuli), smells (lemon, chocolate, and 
spirit stimuli), and combination (containing yellow+sour+lemon, 
chocolate+salted+red, and spirit+sour+green). These groups were 
chosen because the stimulus conditions within the groups share 
a key feature in common. For example, in the color condition, the 
virtual beverage conditions only contained coloring, no electric 
taste stimulation and no additional smell. 

First, the within-group differences are assessed. The differences 
in overall liking, smell liking, taste liking, and temperature liking 


are compared within the different groupings of the stimuli. Re- 
peated measures ANOVAs were conducted to test if there were 
differences in liking within the groups. This was done to ensure 
that the members of each group were not different so that the scores 
within the groups could be collapsed. There were no differences 
within the groups of smell liking and taste liking (p’s > .05). For 
overall liking, there was no difference within the color, electric 
taste, and combination groups. There was a significant difference 
in overall liking within the smell group F(2, 60) = 5.41, p = .007, 
n* = .091. Tukey’s post-hoc analyses revealed that the spirit smell 
is overall liked less than both the lemon smell (p = .015) and the 
chocolate smell (p = .018). 

Similarly, the differences in the perception of individual sensa- 
tions of salty, sour, sweet, and bitter are compared within each 
grouping (i.e., color, electric taste, smells, and combination) of the 
stimulus combinations. Repeated measures ANOVAs were con- 
ducted to test if there were differences in the perception of these 
sensations within their respective groups. There were no differ- 
ences within groups for the perception of salt, sweet, or bitter. For 
the perception of sour, there were significant differences within 
the smell group F(2, 60) = 4.76, p = .0012, n? = .03, and the combi- 
nation group F(2, 60) = 4.44, p = .016, n? = .049. Within the smell 
group, the spirit smell elicited stronger sensations of sour as com- 
pared to both the lemon (p = .02 ) and the chocolate (p = .034) 
smell. In the combination group, the lemon+sour+yellow combi- 
nation was significantly perceived as more sour compared to the 
chocolate+salted+red combination (p = .012). 


5.1.2 Comparing the Virtual Beverage Groups in Terms of Taste: 

Here, the perception of different virtual flavor stimuli groupings 
(color, electric taste, smell, and combination) was evaluated in the 
virtual beverage in terms of four primary taste sensations: salty, 
sour, sweet, and bitter. Initial inspection of data reveals differences 
for salty, sour, and bitter sensations (Figure 4). A repeated measure 
ANOVA revealed a significant difference for the perception of salty, 
F(3, 90) = 8.96, p < .001, n? = .092. Tukey’s post-hoc comparisons 
reveal that the electric taste and the combination groups are not 
different from each other (p > .05) and are both greater than the color 
and smell groups (p < .05). Significant differences in the perception 
of sour were found, F(3, 90) = 14.3, p < .001, n? = .12. Like the salty 
perception, the electric taste and the combination groups were 
higher in sour than the color and smell groups (p < .05). There 
were also significant differences found in the perception of bitter 
F(3, 90) = 9.94, p = <.001, n? = .077. Again, the perception of bitter 
was greater in the electric taste group and the combination group 
as compared to the color and smell group p < .05). There was no 
significant difference between the groups for the perception of 
sweetness. Overall, the electric taste group and the combination 
group showed higher flavor sensations. 


5.1.3. Comparing the Virtual Beverage Groups in Terms of Liking: 

Participants’ overall liking, smell liking, taste liking, and tempera- 
ture liking of the different groupings are compared to understand 
how the different types of stimuli configurations affect the liking 
towards virtual beverages. Initial inspection of the data suggests 
a general liking for the color group compared to the combination 
group (Figure 5). A repeated-measures ANOVA show a significant 
effect of stimulus group on overall liking (F(3, 90) = 5.82, p < .001, n? 
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Figure 4: Average intensity scores of different stimuli groups in terms of primary taste sensations reported from the virtual 
beverage glass (error bars represent standard error). 
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Figure 5: Average liking scores of different stimuli groups using virtual beverage glass (error bars represent standard error). 
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Figure 6: Positive vs. negative emotions perceived by the par- ‘understanding’. The Lemon smell did not have a close association 


ticipants (error bars represent standard error). with any specific emotion. The participants associated the spirit 
smell with a ‘daring’ emotion, which was interesting. Following a 


similar trend, the participants reported different mediated emotions 
when combined stimuli of color, taste, and smell were provided. For 
example, the Spirit smell was associated with ‘daring’ emotion, and 
when combined with the Sour taste and Green color, the partici- 
pants felt ‘active’. On the contrary, the combined stimuli of Lemon 
+ Sour + Yellow, which was intended to imitate a lemonade-like 
experience were related to ‘guilty’ and ‘worried’ emotions. 
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= .10), taste liking (F(3, 90) = 8.00, p < .001, n? = .12), and temperature 
liking (F(3, 90) = 5.69, p < .001, n? = .11). Tukey’s post-hoc anal- 
yses showed that participants overall liked the color group more 
than both the electric taste group (p = .002) and the combination 
group (p = .008). Participants’ liked the taste of the color and the 
combination more than both electric taste and smell groups (p’s < 
.05). Finally, participants liked the temperature of the color group 
more than the electric taste (p = .006) and the combination group (p 
= .004). There was no difference in how the participants liked the 
smell of the groups (p = .055). Overall, participants liked the color 
group the most but also enjoyed the taste of the combination group. 
It is surprising to detect significant differences in thermal-based 
liking when using the virtual beverage glass where there are no 
additional thermal stimuli applied. This can be due to specific color 
and other stimuli combinations and merits future explorations. 


5.2 Results: Virtual coffee mug 


5.2.1 Grouping the Virtual Coffee Mug Modalities: In the same 
way that there were groupings created a priori for the virtual bev- 
erage, groupings for the virtual coffee were created. They are as 
follows: flavor (chocolate, mint, and coffee flavorings only), hot 
flavor (hot temperate + chocolate, hot temperature + mint, and hot 
temperature + mint), cold flavor (cold temperate + chocolate, cold 
5.1.4 Emotion responses and associations with virtual flavor stimuli temperature + mint, and cold temperature + mint), hot temperature, 
in Virtual Beverage Glass: Figure 7 depicts the various emotions cold temperature, and no stimulation. 
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Figure 7: Elicited emotions from the virtual beverage glass” 


The groups, flavor, hot flavor, and cold flavor were examined for 
within-group differences regarding the primary sensations (salty, 
sour, sweet, and bitter) and liking scores (overall, taste, smell, and 
temperature). Initial inspection of the data suggests that overall 
there are few (if any) differences within the group. Repeated mea- 
sures ANOVAs confirmed this. Within respective groups, there 
was no difference in terms of the primary sensations (p > .05), and 
for the liking scores, there was only one difference. There was a 
significant effect of smell liking within the hot flavor group, F(2, 
54) = 4.06, p = 0.023, y? = .062. The hot+coffee stimulus had greater 
smell liking than the hot+mint (p = .022). Other than participants 
liking the smell of the coffee when paired with a hot temperature 
more than the smell of the mint, there were no differences within 
the groups. 


5.2.2 Comparing the Virtual Coffee Mug Groups in Terms of Taste: 

The virtual coffee mug groups (flavor, hot flavor, cold flavor, hot 
temperature, cold temperature, and no stimulation) were compared 
in terms of the primary sensations: salty, sour, sweet, and bitter. 
Repeated measures ANOVAs show no significant difference (p's > 
.05) between the groups in any of the primary sensations suggesting 
that the different stimulus components did not alter the primary 
sensations. However, it should be noted that the participants rated 
the groups low on each of the sensations. There is likely a floor 
effect here, where no significant difference can be found because 
the current virtual coffee mug device does not produce adequate 
levels of the flavor sensations. 


5.2.3. Comparing the Virtual Beverage Groups in Terms of Liking: 
Similarly, the virtual coffee mug groups were compared in terms 
of overall liking, taste liking, smell liking, and temperature liking. 
Repeated measures ANOVAs show no significant difference (p’s 
> .05) between the groups for these liking scores. On average, the 
liking scores were above the midpoint, suggesting that participants 
liked the virtual coffee. This is likely due to the device not creating 
strong enough of a difference in the sensations. 


5.2.4 Emotion responses and associations with virtual flavor stimuli 
in Virtual Coffee Mug: Figure 8 depicts the various emotions elicited 
by different virtual flavor stimuli delivered via the virtual coffee mug 
on the correspondence analysis factor map. The axes explain 38.54% 


>See supplemental material for larger image 
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Figure 8: Elicited emotions from the virtual coffee mug’ 


of the variability in the data. The F1 axis displays the emotions 
related to ‘disgusted’ and ‘bored’, while the F2 axis extends between 
positive (good) and negative (guilty) emotions. 

As explained, there is a diverse range of emotions elicited when 
delivering virtual flavor sensations from both apparatus. The emo- 
tions elicited from the virtual coffee mug were closely attached 
to different stimuli when compared to emotions elicited using the 
virtual beverage glass. It is interesting to notice that the hot stimuli 
were not enjoyed when delivered to the participants as it was re- 
lated to ‘bored’ emotion. However, when combined with smells, the 
participants’ responses were varied. For example, hot + chocolate 
stimuli were associated with ‘daring’ emotion, while hot + coffee 
was related to ‘worried’ emotion. A possible reason for this can be 
the overheated Peltier element, which could be optimized in future 
studies. On the contrary, hot + mint stimuli were associated with 
‘peaceful’ emotion. 

The cold stimulus was associated with several emotion clusters 
when simulated alone or in combinations. The cold stimulus was 
related to ‘good’ and ‘pleased’ emotions, while cold + coffee was 
related to ‘good’, ‘good-natured’, ‘peaceful’ emotions. Similarly, 
cold + chocolate stimuli were perceived to be rich in ‘understand- 
ing’, ‘interesting’ and ‘friendly’ emotions, and cold + mint stimuli 
were closely associated with ‘whole’, ‘nostalgic’, ‘steady’, and ‘free’ 
emotions. The mint smell only stimulus was associated with ‘polite’ 
and ‘adventurous’ emotions. A similar trend was observed with 
coffee and chocolate smell stimuli, which were associated with 
‘mild’, ‘loving’, ‘joyful’, ‘pleasant’, and ‘warm’ emotions. Almost all 
the stimuli given in the virtual coffee mug were related to positive 
emotions, which ultimately represents a good user experience with 
not only the apparatus but also virtual flavor sensations. 

The positive and negative composite scores are compared be- 
tween the stimulus groups for the virtual coffee mug (flavor, hot 
flavor, cold flavor, hot temperature, cold temperature, and no stim- 
ulation). There was no significant difference between the groups 
for either positive emotion (F(3, 81) = 2.19, p = .096, n? = 04), or 
negative emotion (F(3, 81) = 1.90, p = .136, n? = .04). Scores for 
both positive and negative emotions were low; thus, it is likely that 
drinking from the virtual coffee mug did not evoke a strong enough 
emotional response on it’s own. 


5.3 Results: Comparison Between the Virtual 
coffee mug and Virtual beverage glass: 


The virtual beverage and the virtual coffee apparatus were com- 
pared to understand differences in how participants liked and rated 
the two different apparatus as well as their emotional response to it. 
Initial inspection of the ratings for the primary senses, bitter, sweet, 
sour, and salt, show differences between the virtual beverage and 
the virtual coffee experiences. Independent samples T-tests reveal 
that the participants overall rated the virtual beverage higher than 
the virtual coffee on bitter (t(58) = 2.78, p = 0.007, d = .71), sweet 
(t(58) = 2.13, p = 0.001, d = .55), sour (t(58) = 3.41, p = 0.001, d = .88), 
and salt (t(58) = 3.91, p < .001, d = 1.01). These results show that 
participants felt that the virtual beverage had stronger flavors than 
the virtual coffee in every way. Overall, the data seems to show that 
participants liked the virtual beverage more than the virtual coffee. 
Independent samples T-tests show greater overall liking (t(58) = 
2.30, p = 0.025, d = .60) and temperature liking (t(58) = 2.30, p = 
0.034, d = .56). There was no significant difference for neither the 
taste nor the smell liking. It is likely that the participants liked the 
stronger flavor profiles of the virtual beverage glass. 

The participants’ positive and negative emotion composite scores 
are compared between the two apparatuses (Figure 6). Independent 
samples t-tests show that there are no differences between virtual 
beverage and the virtual coffee for both positive emotions (t(58) = 
1.70, p = 0.093, d = .44), and negative emotions (t(58) = .60, p = 0.554, 
d = .15). Exploring the Cohen’s d effect size for positive emotions, 
there is an evidence of a medium effect for positive emotions where 
the virtual beverage elicited higher positive emotions than the 
virtual coffee. 


6 DISCUSSION AND CONCLUSION 


The results of our user experiment with both apparatus revealed 
that the participants 1) perceived various flavor sensations when 
presented with different stimuli configurations, 2) generally liked 
different virtual flavor sensations, 3) distinguished different ba- 
sic taste sensations as per the stimuli, and 4) associated different 
emotions for virtual flavor sensations. One significant finding that 
validates our approach is that the participants have liked all of 
the stimuli configurations, in general, and some more than others. 
This finding is encouraging as the way virtual flavor components 
are delivered differs from a traditional drinking experience where 
the volatile aromatic compounds are released retronasally through 
mastication inside the mouth. In our approach, smells are released 
orthonasally while the electric taste is only applied to the tip of the 
tongue, and thermal stimuli only applied to the outer lips. 

Our approach of delivering flavor sensations and their specific 
combinations is novel and a first-time experience for all the par- 
ticipants, this might have an influence on their liking scores. The 
“Neophic effect” or the fear of trying new things in the controlled 
experiment could have affected the liking scores perceived for each 
stimulus and apparatus. Furthermore, 1) the two apparatus devel- 
oped is a little heavier and different from everyday utensils, and 
2) the stimuli configurations are not calibrated towards individual 
preferences may also negatively affect participants’ responses. 

The responses received for sweet taste was evaluated to be weak 
in that there were no differences between the groups via the stimuli 


delivered from the virtual beverage glass. As expected, there was 
an increased sensation of salt, bitter, and sour for the electric taste 
group, especially compared to the color group. This was also true 
for the combination group, which included an electric taste compo- 
nent. The responses were low for the color and smell group. This 
may indicate that a large factor for altering taste is the electric taste 
component, or that the smells provided were not strong enough. 
It also shows further opportunities to optimize the stimuli config- 
urations to achieve the expected ‘just about right’ perception for 
all the stimuli. Interestingly, the color and smell groups were liked 
more in terms of overall liking suggestions. In contrast, the electric 
taste and combination groups had a stronger flavor profile, however, 
with lower liking scores for the flavors. Regardless, participants 
liked all of the flavor combinations for the virtual beverage. 

A better replication of coffee would enable a better assessment 
of the emotional effects of different coffee flavors and temperatures. 
One method to increase the flavor intensity would be to introduce 
a taste input to occur with the smell. While it is true that there is an 
existing taste-smell association, and odor may be enough to elicit 
flavor perception [12], taste and smell combinations can increase the 
intensity of both the smell and the taste when presented together [5]. 
Also, the way that an odor is presented can alter the impact that 
the odor has on a flavor sensation. When an odor is presented 
retronasally (in a consumed solution), it is more identifiable than if 
presented oronasally [45]. The ability to precisely adjust the tastes, 
smells, and temperatures of the beverages give this research the 
unique ability to investigate precisely how these different modalities 
play a part in an overall flavor experience. Another way to improve 
the virtual coffee would be to improve the cup design to better 
replicate a coffee cup (more realistic coffee cup shape may cause 
the participants to perceive more significant flavor sensations). The 
beverage vessel can impact the taste of the beverage itself [23]. 
The shape, color, and texture can affect the smell, sweetness, and 
perceived acidity of the coffee [7, 8, 48]. 

The elicited emotions from different virtual flavor sensations 
were generally positive. It is also observed that the emotions re- 
ported from the virtual beverage glass were recognized as calm, 
peaceful, and pleasant emotions while from the virtual coffee mug 
were identified as energetic, happy, and merry. These emotions are 
generally associated with soft drinks and coffee in our everyday 
lives [4, 49]. None of the stimuli were related to disgust or anger. 
These findings confirmed that the participants enjoyed most of the 
stimuli configurations. Future research should also focus on even 
more smell and color sensations, developing seamless apparatus, 
and various base drinks instead of plain water. When comparing 
the two apparatuses, a few key points should be noted. First, the 
virtual beverage elicited stronger taste sensations than the virtual 
coffee. Second, participants liked the virtual beverage more than 
the virtual coffee. It is likely that there was not a strong enough 
flavor sensation for the virtual coffee. People typically expect that 
coffee has a relatively strong flavor and drinks more similar to the 
virtual beverages can have less flavor and still be considered ac- 
ceptable. Finally, there was a trend for the virtual beverage to elicit 
greater positive emotions than the coffee. This may be due to a 
larger discrepancy between the virtual coffee’s expected flavor and 
the actual flavor. A disconnect between what an individual expects 
and experiences may lead to less positive emotions [44]. 
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